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ABSTRACT 

This study describes two surveys of the users of the 
Public Documents and Maps Department of the Duke University Library. 
The information gathered in the course of this inquiry is reported: 
(1) in a profile of the users of the department's reference service, 
and (2) in a comparison and analysis of the evaluations of reference 

:encounters by both users and librarians. The results of the first 
survey, the user. prof ile, reveal that most users of the documents 
department are Duke University students seeking information in the 
area of the social sciences. The types of questions most frequently 
asked are requests for legislative information, a specific title or 
article, an ^ statistical information. Results of the second survey, 
which focus, on the effectiveness of the department's reference 
service, indicate that: (1) many users may Irave an inadequate 
knowledge of government documents., (2) users tend to express high 

'rates of satisfaction with the outcomes, of reference encounters, and 
(3) users seem to respond favorably to instruction by librarians in 
the use of the department's reference tools and sources. The 
questionnaires used for both surveys, notes, and a bibliography are 
appended. (THC) 
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ThlL study describes two surveys of the users oi the Public Docu- 
ments and Maps Department of the Duke University Library. The information 
gathered in the course of this inquiry consists of a profile of the users 
of the Department's reference service and a comparison and analysis of the 
evaluations of users and librarians about a series of teference encounters. 

The results of the first survey, the user profile, reveal that most 
users of the Documents Department are Duke students seeking information in 
the area of the social sciences. The most frequently asked types of 
questions are 1) legislative informatlon» 2) request for a specific 
title or article, and 3) statistical information. 

The results of the second survey indicate that 1) many users may 

have an Inadequate knowledge of government documents, 2) users tend to 

t. 

express high rates of satisfaction with the outcomes of reference 
encounters, and 3) users seem to respond favorably to instruction by 
librarians in the use of the Department's reference tools and sources. 
Headings: 

Reference services — Evaluation 
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College and university libraries — reference services 
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INTRODUCTION 

Bernard Fry, in a 1978 report for the National Conmiisslon on 

Libraries and Information Science, notes that government documents are 

probably used less than any othe^. kind of publication in libraries, and 

that government documents departments often lack the nece^?8ary resources^ 

and staff to utilize their collections effectively. In light of these 

problems he write?: 

It is significant that several major academic and research 
libraries, as at the University of California, feerkley, have 
recently made a comprehensive re-examinatlon of policies, pro- 
cedures, collections, and services affecting their government 
documents departments j This kind of study is long overdue 
at many libraries.... 

Other authorities on government publications have also emphasized 

the need to stuidy the services of documents departments. In a recent 

study of reference service provided by selected documents departments 

in academic libraries, H^irnon and McClure point out: 

The first step in improving a service, such as the reference 
process in academic depository libraries, is to investigate, 
describe, and understand the current situation and existing 
fatetors related to the effectiveness of that service. ' 

In view of the attention focused on studying the problems of 
documents departments in academic libraries during recent years, the 
author of this study decided to examine the reference service ol the Public 
Documents and Maps Department of Perkins Library at Duke University. Like 
the documents departments referred to by Fry and like some of those studied 
by Hernon and McClure, the Duke Public Documents and Maps Department con- 
tains a large collection (about 600,000 Items) of U. S. Government docu- 
ments and serves a major academic community (about 9,000 students and 1,400 

0 



faculty) as well as the general pubTic. 

However the Public Documents and Mapn Department (hereinafter re- 
ferred to as the Documents Department) currently lacks sufficient inform- 
ation about Its reference service to perform the kind of evaluation called 
^or by the writers cited above. Wliile the staff of the Department keeps 
simple counts of the numbers of reference and directional questions, they 
need more data about the department's reference service in order to 
evaluate and improve it. 

For example, data are needed about the users of the reference se^'^ice: 
such as who they are^, what kind of information they are seeking and how 
they are referred to the Documents Department. Infonhation is also needed 
about the referencie process: for example, the extent to which the users 
and librarians are satisfied with the results of reference encounters. 

Therefore, the purpose of this study is to collect information 
from both users and staff members of the Documents Department as a first 
step In the evaluation of the Department's reference service. The informa- 
tion gathered in the course of this inquiry consists of a profile of the 
users of the Department's reference service and a comparison and analysis 
of r.he evaluations of users and librarians about a series of reference 
encounters. 

This study is significant because, while many other such studies have 
been repprted for regular reference departments, there have been few reported 
studies of reference service In government documents departments. 

The first chapter of this study examines the many approaches to 
measuring and evaluating reference service. This section discusses some of 
the major studies in each category and po-'.nts out their comparative 
advantages and disadvantages* It also reviews related studies In the area of 
government documents. 



Chapter II outlines the defilgn of the two surveys used In this 
study. The results of these surveys are presented In the third chapter. 
The final chapter presents conclusions and recommendations based on the 
results of this study. 



CHAPTER I 

METHODS OF MEASURING AND EVALUATING 
• REFERENCE SERVICE 

Twenty years ago Samuel Rothsteln reviewed the literature on the 

evaluation of reference service and concluded: "The measurement and 

evaluation of reference service. has been more often discussed than 
3 ' ' ' ' 

attempted*" 

Ten years after Rothsteln^s article, Terry Weech also reviewed the 
literature on this subject and found that library researchers, perhaps 
spurred In part by Rothsteln *s article, were making more efforts to 
actually carry out evaluations.^ 

Today librarians continue to show a strong interest in this topic, 
and there have been many Important studies reported in the literature 
over the past decade. However, despite such strong interest in this 
topic, there is no single, universally accepted methodology for the 
evaluation of reference service, A review of the literature on 
evaluation reveals many different styles. 

thus, for the sake of clarity, this section categorizes and briefly 
discusses various methods of evaluation as follows:^ 

- Enumeration and Classification of Reference Questions 

- Evaluation of Reference Collections 
,^ Evaluation of Reference Personnel 

f - Cost Analysis 

- Comparison to Reference Service Standards 

- Administration of Test Questions 

- Unobtrusive Testing 



Before discussing the above methods, It would be useful to clarify 
the meaning of two term^?, — measurement and evaluation. 



Measurement vs. Evaluation 

♦ 

Rothsteln mfide a distinction betwejen what he cbns'ldered true 

evaluation and mere measurement. "Measurement," as defined by Rothsteln, 

Is "description In quantitative terms;" whereas "evaluation" Is "the 

6 

rating or- assessment of worth*" In addition, Rothsteln noted: "Evaluation 
presupposes measurement against a specific standard or yardstick or 
goal. . . ."'^ . . { 

WhHe the above distinction seems valid, other researchers do not 
adhere to such a strict definition of evaluation. Thus this study will 
use a broad definition of evaluation.j^ In general, evaluation of reference 
service means any attempt to ^assess its effectiveness,; whether or not there 
is comparison to a specific standa^-d. Also, measurement may be viewed as 
an Integral part of evaluation of reference service , since the first and 
other steps of the evaluation process usually Involve description In 

qiAantltativc terms r 
it 

Enumeration and Classification of^ 
Reference Questions 

The simple counting of ref^erence questions is the most common 
measure of actlvitiy.. at the reference desk. By Itself such a. count gives 
nb indication of the effectiveness of reference service. 

The classification of reference questions; by type (e.g., ready- 
reference or in-depth search), by subject, by type of user or other 
categories goes a step beyond a gross count, and it is more meaningful. 
The importance of knowing who the users arc and what questions they ask 
seems obvious: for example, such classifications can be used to aid in 



er|c 



planning the reference collection, to guide staff training and to compare 
one reference department to another. ' . 

While there are no current and reliable .estimates of how many ^ 
11^ raries count ^nd classify reference questions, a. 1977 study by 
Marcella Cuickl reported on the most common kinds of information gathered 
about reference servtce. Cuickl wrote that an Informal American Library 
Associatipn survey revealed that, the most frequentfy collected cate- 
gories of statistical ^data on reference were as follows: 

1) . Type of reference question (e.g., ready reference or In-depth 
search). 

2) Form of request (In person, by telephone, or letter). 

3) Directional questions. 

A) Time of day when questions were asked. 

5) Type of sourc^e used (t.g. , refkirence book or index), 

6) Type of user (e. g. , student or faculty member). 

Other categories included the length of time to answer, subject 4rea of 
questions and referral of questions to other librarians and agencies. 

. While collecting such information may seem straightforward at first- 
glance, there are problems relating to the reliability and validity of 
the counting and classification of question^. For examnple, staff 
Inattentiveness in recording transactions may Ifead-to unreliable totalr 
of questions. Also, staff members may miSsunderstand their library's 
question classification scheme, thus leading to invalid results. 
Nevertheless, if carried out carefully, this method can give librarians 
and administrators meaningful feedback about what goes on at the 
reference desk. 



^ Evaluation of Reference Collections 

it' 

/• This category of evaluation usually Involves the comparison of a 

llbrarjr's reference collection to standi^rd book lists and bibliographies • 
f * 

Weech* cites two studies which used more Innovative techniques than 
comparison to-standard lists. A study by Houser analy?.ed the currentness 
ot a reference collection by counting tKe distribution of copyright dates 
of the^materlals. Another study, conducted by the New York State 
Educaeion Department, sought user opinion on th6 usefulness p^ the refer- 



*ence collections in a sample of public libraries, v 

' \ ■ ." ' 

Evalua^tlon of Reference Personnel 

.. . — 

This type ofF^study may'^^nvolve any of seve^'al different approaches. ^ 

I f 

Sc \ studies focus on the nuj^ber of reference emplQyees relative to the 

total staff, or on the proportion of total staff time spent at the refer- 
''..ice desk. For example, a 1972 study of^ the; Columbia University library 
system by Mount and Fasana, reported that 11 percent of total staff time 
Involvedl^ef erence wortv. ^ . 

Other researchers f^cus more directly on reference personnel by 
observing them during the refefepce process. For example, Bunge observed 

selected member^ of the reference staff in nine* public , libraries and compared 
* t 

the performance of professionals and nonprofessionals. He found that the 
professional and nonprofessional staff members answered * questions with about 

the same degree of accuracy, ^ut that th^ professional i answered questions 

12 • . ' ' A 

more quickly and ef f icl^ently. 

A 1976 study by Benlta Howell and ot^^Ys analyzed group differences 
between the evaluations of users and librarians about a series of reference 
^ncoynters. Among the Hpwell study findings were: 
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1) There was a high degree of user eatisracticn with the librarians' 
performance. Eighty-four percent of vAets said the librarians had pin- 
pointed their information needs "very well," and eighty-^seven percent felt 
that the librarian had supplied "about the right anount of Information." 

2) There* was a statistically sig.vif leant difference, how<2ver, between 
the users • ratings of the librarians and' the librarians^ ratings of 
themselves: in general, libravians rated their own performance lower 

that the users did. 



\ 



[The authors of this study concluded;] \ 
This finding suggests two things: (1) that patrons' expect- 
ations for service are considerably more modest^^han librarians* 
performance standards, and/or (2) that patrons haVe difficulty 
distlngulching between their pleasure at being helped, and their 
satisf^'xtion with the quality of help being received* 

Another uignlflcuAf study In this category was reported by Mary Jo 

Lynch in a 1978 article in The Library Quarterly . She observed and 

♦ • ■ • 

recorded the actions and words of reference librarians during over 300 
reference Interviews in four Kew Jersey public libraries. According to 
Lynch, "ft]he primary objective of the investigation vc.j t-o give form 

15 

and structure to the phenomenon known as the reference Ittterview. . . ." 

The* results of her study belied several commonly held beliefs about 

! ■ 

the reference interview. Among her conclusions were the following: 

1) Not all reference transactions Involve a reference Interview 
(only about half of the transactions In the Lynch study Involved an 

Interview),, . 

■ ' — " <-i 

,2) "Closed" or specific questions, rather than "open" or non- 
specific questions may be a more appropriate strategy for the reference 
librarian to follow in the reference Interview. Although it is commonly 
assumed that open questlcns are better than closed questions, Lynch found 
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that ninety percent of the librarians* questions in her study were closed 
She speculates that the typical environment at a reference desk is not 

suitable "...for the self-revealing, tii.;4i-consumlng process of responding 

16- 

to open questions." 

3) The reference interview is not similar to interviews conducted 
by other professionals such as doctors or lawyers. This conclusion is 
based on Lynch's finding that :vne typical reference interview Involved 
the asking of only one or two "primary" questions (i.e., "...questions 
through which the librarian introduces some aspect of the patron's 
search for information and asks for content which is new to the 
interview."^^ ); whereas a typical medical interview may involve up to 
AO primary questions. 

4) Contrary to the common assumption that library users do- not ask 
for what they actually want. Lynch found that "...in many cases patrons 
do ask for what they want although not in as much detail as is necessary 
for the librarian to be able to help."^^ 

In sum, Lynch's study provides a useful framework for further 
analysis of librarians' behavior during the reference interview. 

Cost Analysis 

Several investigations have sought to calculate the average cost of 

Individual reference encounters., —For example , a^-stiidy-by-Falmour and 

Gray of seven Illinois public libraries computed the direct labor costs 
for answering different types of reference questions (e.g., simple faet 
or bibliographic citation). ' This study and other similar ones, 
however, ig^hore overhead costs such as library materials. Moreover, most 
library cost studies do not attempt to compare costs to the benefits 
received by the users (which would be difficult to assess objectively). 
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Thus the usefulness of cost analysis as a method of evaluation Is 
questionable. 

C omparison to Reference Service Standards 

In 1976 the Reference and Adult Services Division of the American 

' Library Association Issued "A Commitment to Information Services: 

20 

Developmental Guidelines," While this brief document Is helpful as a 

gen»**-al guide to what services and functions are to be included In a 

reference department, it contains no specific quantitative standards 

which could be used as a basis for evaluation. 

In contrast, a set of standards that does contain quantitative 

guidelines is the "Standard's for Reference Service in Public Libraries," 

21 

written in 1970 by the Library Association of Great Britain. It 
recommends minimum standards for the size and subject composition of the 
reference collection, the types of services" to be offered, physical 
facilities, and the qualifications of reference personnel. For example, 
the standards recommend providing one seat for every 500 members of the 
library* s population and 25 square feet of floor space for each seat. 

They also recommend a reference collftction of at least 200 volumes per 

22 

1000 population and a ratio of staff to population ot 1:20,000. 

While such standards are not directly applicable to all libraries 

-In the -United 5 tea > they do suggest that it is feasib^le for groups of 

similar libraries, for example the Association of Research Libraries, to 
come up with quantitative reference standards based on their own criteria 
and needs. Such quantitative standards would be more useful than the 
current ALA guidelines, which are too general* 

Administration of T est Questions 
This method attempts to evaluate the accuracy of the responses of 

ERJC 7. ' 
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reference personnel to questions requiring an objective answer. The method 

usually Involves the preparation of a list of test or sample questions which 

are "...selected fo provide a range of difficulty and require a variety of 
23 

sources." Test questions were first used in an extensive way by tlie 

New York Committee on Public Library Service in a 195? study of a sau.ple of 

public libraries. 

However, those and other such studies during the 1960'8 suffered from 
an obvious weakmjss: the test subjects knew they were being observed and 
evaluated. Under such conditions, according to the well known "Hawthorne 
effect," a librarian's behavior is apt to change. He or she may search more 
carefully for an answer and so perform better than under normal conditions; 
conversely, the librarian may feel pressured by the test conditions and his 
or her performance may deteriorate^ 

As Lancaster notes, the solution to this problem lies in the use of 
unobtrusive testing: 

Ideally, it would be preferable to administer a controlled test 
with the subject unaware that he is being studied. Such a test is 
likely to be more satisfactory In many ways than an obtrusive test, 
because it could measure the performance of the reference librarian 
under actual working conditions rather than under the artificial 
conditions of an obtrusive study. 

Unobtrusive Testing 
While there are also problems associated with unobtrusive testing, 
this method has been used successfully in several major studies over the 
last 15 years, including one concerned with reference service for 
government documents. Today unobtrusive testing is the major trend In 
the evaluation of reference service* Because of Its Importance, this 
section will briefly review the major studies which have used this method 
and the method's advantages and disadvantages. 
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Procedures Use d in Unobtrusive Testing 

All of the authors of the major stuales followed approximately the 
same procedures In carrying out their research: 

1) They selected test questions of a factual o, ready-reference 
nature. 

2) The researchers pre-tested the questions and discussed them 
with other librarians to ensure that the questions were typical for a 
given library. 

3) A random sample was selected from a population of libraries, 
e.g., public libraries in New Jersey or academic libraries in the 
southwestern United States. 

A) The authors trained proxies, usually students, to administer the 
questions in person or by phone to the reference staff of the libraries. 

5) The authors evaluated the accuracy of the answers and analyzed 
the results in relation to several variables, e.g., the size of a 
library's collection, the type of Institutions or the difficulty of* the 
question. 

The Crowley and Childers Studies 

The first two major studies to use unobtrusive testing were those 
completed by Terence Crowley in 1968 and by Thomas Childers in 1971. 
The studies were published together In Information Service in Public 
Libraries . 

Crowley's study investigated whether libraries with high per capita 
expenditures would perform better than those with low per capita 
expenditures* While he found no statistically significant difference 
between the two categories, his other findings were enlightening. For 
example, the composite percentage of questions answered correctly by all 
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libraries was only percent. In addition, many librariei? performed 

most poorly in locating answers to questions requiring currert information, 

Childers* study and results were quite similar to Crowlay's, with 
the exception that all questions were administered by phone. Childers 
also found no significant relationship between correct responses and per 
capita expenditures; the overall percentage of correct responses was 
55 percent. 

The main value of the Crowley and Childers studies was that they 
demonstrated the value and feasibility of unobtrusive testing, and they 

stimulated the interest of other researchers in this technique. 

\. 
(■ ■ 

. \- 

The Suffolk .County Test \^ 

In 1977 Childers was Invited by the Suffolk (N.Y.) County Cooperative 
Library System to perform a large study using unobtrusive testing of 57 of 

the system's libraries. This s tudy' was important because \it was done at 

■ \ 

the invitation of a library system which sought to use unobtrusive testing 
as a means of Improving its reference service. Childers also performed 
extensive testing cf outside agencies (e.g., a government agency or another 
library) to which the proxies were referred when the original library could 
not answer the question. 

Again the overall percentage of correct answers was only about 50 per- 
cent. In addition, Childers found that when a reference staff member failed 
to provide an answer, he or she referred the proxies to other agencies only 
about half the time; however, when the proxies followed up on those referrals, 
they received a correct response for 67 percent of those questions." Childers 
concluded that the libraries in this study needed to improve their policies 
'on referral of difficult questions, and that reference librarians in general 
need tc expand their use of sources beyond the printed sources, of the 



traditional In-house reference collection. 

Unobtrusive Testing In Academic Libraries 

Other library researchers soon applied the to«chnlque of unobtrusive 
testing to reference departments In academic libraries. Two studies of 

6. 

telephone reference service, one by Marcla J. Myers and one by Jassim M. 

28 

Jlrjees were published In 1983. Myers tested a sample of 40 colleges 
in the southeastern United States and Jlrjees used a small sample of 5 
colleges In the northeast. 

The overall results for both studies vere remarkabljr similar to the 
results of the public library tests. Myers' libraries ansvered about 50 
percent of the questions correctly and Jlrjees' libraries about 57 percent. 

Both authors also sent questionnaires to the libraries in their 
respective samples. One Important finding in both stucies was that few 
of the reference departments had concrete written policies for Reference 
service. This fact may account for some of the Inconalptency the authors 
encountered In response to the test questions (e.g., the reference staff 
at some libraries gave out conflicting information concerning the library's 
policy towards telephone reference service). 

The Hernon-McClure Study 

Another unobtrusive test of reference service In academic libraries 
focused exclusively on reference service for government documents. This 
study, entitled Improving the Quality of Reference Service for 
Government Publications by Peter Hernon and Charles McClure, tested 
documents departments of academic libraries in the southwest and 
northeast. 

The overall percentage of correct responses was 37 percent (49 percent 
in the northeast and 20 percent In the southwest). After analyzing several 



15 

Institutional variables (such as size and budget of the library and organ- 
ization of the documents collection), the authors found no single variable 

that had a strong correlation with the percentage of correct answers at a 
29 

given library. 

Based on these findings and their observations of reference staff during 
the tests, Hemon and McClure conclude that the competency and attitude of 
"...the individual library staff member is the single most important factor 
affecting the quality of reference service for govei^nment documents. "^^ 
Furthermore, they recommend that: "Concentrating on the skills and 
competencies of individual library staff members may well upgrade the quality 
of reference service,,,." The authors' recommendations will be discussed 
further In relation to the Public Documents and Maps Department at Duke in 
* the conclusion of this study. 

The Pros and Cons of Obtrusive Testing 

While more study Is needed, the results of the studies described 
above suggest a few. Important points about the provision of reference 
service In both public and academic libraries. Most Importantly, a user 
has only about a 50 percent chance of finding the correct answer to a fact- 
ual question at the reference desk of both public and academic libraries- * . 
This percentage is probably significantly lower than what the typical refer- 
ence librarian would Imagine his or her own performance level to be. Thus 
as Lancaster writes, those studies should serve as a warning flag to members 
of the profession to examine and evaluate further the reference process: 

The greatest value of these types of analyses, lies in their 
diagnostic possibilities. Evaluation procedures need to be 
applied intensively to individual libraries, or to the libraries 
in a particular system, to identify weaknesses and sources of . 
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failure and to lead to corrective actions designed to Improve 
future performance. Such corrective actions could take the form 
of Improving procedures for selection and training of staff i im- 
proving the reference collections in specific areas. ..or changing 
practices in the reference division (e^g.^ establishing new files, 
routine clipping of newspapers, assig^^ng responslbllty for keep- 
ing various reference tools current). 

However, there should not be an overractlon to the results of the 
unobtrusive tests. For these studies test only one aspect of refererxe 
service — the response to factual questions. Other areas of referenc^ ser- 
vice, such as individual guidance for in-depth subject searches and biblio- 
graphic Instruction are equally important, Furthf-irmore,. considering that 
most users would probably have a much lower chance of finding the correct 
answers themselves, perhaps a 50 percent correct ratio is not unexpected, 
In sum, unobtrusive testing has proved to be an effective method of eval- 
uatlng reference service, and it probably will become a routine technique 
In many future programs of evaluation. 

Other Research on Government Documents 
Thus far the only study to specifically address the problem of 
measuring and evaluating reference service for government documents is 
the Hernon-McClure study. These authors also provide a useful review 
of other recent lesearch on government dov^uments. While such a review is 
beyond the scope of this paper and seems unnecessary to repeat, it would 
be appropriate to mention a couple of areas of research on government 
documents which are related to reference seirvice. 

There have been a few Important use studies of government documents. 
For example, Hernon surveyed usage of government documents by faculty members 
in thexsoclal sciences at a group of colleges, and found "...a wide range of 
use and nonxuse;'* and he noted that "...a small portion of the government 

N 
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publication collection accounts for the vast majority of use." 

A recent study of government documents use at Miami University 
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of Ohio reported flimllar findings: "Documents in ? lew areas of the 
collection account for a large percentage of use." The study also 
revealed that only about three percent of the print collection and one 
halt of one percent of the microfiche collection was used during the year. 
?Tie authors conclude that there is a need for a more selective collection 
development policy as well as a need to improve user education and outreach 
activities* 

According to Hernon and McClure, other research related to 
reference service for documents has shown that: 1) government 
periodicals are rarely represented in traditional periodical indexes; 
2) government documents are often not well integrated into overall 
library administrative, reference and collection development processes ^ 
and; 3) documents libraries have been unable to keep pace with new / 
technologies used in other reference work, e.g., online database 
searching and in-house automated systems. 

Summary \ ■ 

' Recent years have seen an Increase In interest and activity among 
N library researchers related to the measurement and evaluation of 
reference service. While researchers have used many methods an4 
approaches, unobstruslve testing, despite its limitations has emerged as 
the most significant method of evaluation. As In the Hemon and McClure 
Study, it has proved to be a valuable tool In highlighting weaknesses in 
reference service. While other research on reference service, for 
government documents is limited, studies tend to show an 
under-utilizatlon of government documents and a lack of integration of 
documents into the mainstream of service. 
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CHAPTER II , . 

SURVEYS OF THE USERS OF THE 
PUBLIC DOCUMENTS AND MAPS DEPARTMENT 
Chapt er^ I has provided an overview of the methods of measuring and 
evaluating reference service. This chapter focuses on the provision of 
reference service at. the Public Documents and Maps Department of Perkins 
Library at Duke University (hereinafter referred to. as the Documents 
Department). 

This study used a two-stage survey procedure for the purpose of 
gathering Information about the Dofcuments Department. The first survey 
profiles the characteristics of the Department's users, while the second 
survey attempts to assess the effectiveness of the Department's reference 
service. • * \ 

o 

Research Questions and Hypotheses 
Because *the aim of the first survey was to generate baseline data about 
the users of this service, formal hypotheses were neither developed nor 
tested; Instead th5 following research questions guided this phase of the 
study: '~ 

1) Who are the primary users of the Documents Department^ in what 
subject areas are they seeking information; and what type of questions do 
they ask? 

2) What proportion of the patrons are using the Department for the 
first time? 

3) What proportion of the users are referred to the Department and 
by ,whom are they referred? 

A) How much time do patrons spend in the Departnent on a single visit? 




19 

5) Are there any differences In the Information-seeking patterns of 
major groups of users, i.e., undergraduate students and graduate students/, 
faculty? 

As will be seen in the following chapter, the analysis of the <iata 
generated in this phase of the study supports a major assumption of the 
study, i.e., that the users of the Do'cuments Depar^ment have in common 
certain characteristics and vinformatlon-seeking patterns. For example, a 
large majority of the users are seeking information In the subject area of 
the sociel sciences. 

The second survey focuses on a series of reference encounters between 
users and librarians. The goal of this phase of the study was to attempt t 
assess the .effectiveness of the Department's reference service by comparing 
the users' and librarians' evaluations of major aspects of the reference 
encounter, 

Howell notes the significance of examining both users' and librarians' 
viewpoints: 

Since the reference encounter is an interplay between the patron 
and the librarian. . ^ it seemfs] essential to Investigate the 
librarian's feelnga.about each encounter as well as the patron's 
' feelings about it. 

In addition, since user surveys generally show high levels of patron 
satisfaction with librarians' performance,''^ the librarians' evaluations 
„^s_i_standard of comparison for users' evaluations. 
The general hypothesis for the second survey was that there would be 
statistically significant differences between users and librarians In their 
luatlons of the reference encounter. Using this hypothesis jas a guide, 



the following specific' hypotheses are stated in the null form, j 

There is no statistically significant difference between ratings 
by users^nd librarians of the user's knowledge of government documents. 
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There is no statistically significant difference between. evalua- 
tions by users and librarians of the librarian's ability to pinpoint the 
user's information needs. 

There is no statistically significant difference between evalua- 
tions by users and librarians of the ability of the librarian to supply the 
right amount of information to the user. 

There is no statistically significant difference between users and 
librarians in their awareness of intended instruction of users by librarians, 



Definition of Tenns 
The meanings of certain terms used in this stud\ are to be understood 
as follows: 



Referefice question, 
encounter or transaction 



Government document 
or publication 



Reference librarian 
or librarian 



User or patron 



"...[A]n Information contact which Involves 
the use, recommendation, interpretation, or 
Instruction In the use of one or more 
information sources or knowledge of such 
sources^oby a member of the reference..* 
staff. "^^ 

"Informational matter which Is printed as 
an individual document at government ex** 
pense or as required by law." 

Any of four staff members of the Documents 
Department at Duke who holds an MLS degree 
and was involved in the second survey. 

Any person including, faculty, staff, or 
students at Duke or the general public 
who addresses a reference question in 
person or by phone to a staff member of 
the department. 



Assumptions ? 
In carrying out this study, the following assumptions were made: 
1) The users of the Documents Department share certain distinctive 

charactei*ietics and information-seeking patterns, and It Is possible to 

measure selected of these variables. ^ 
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2) The survey Is a valid and reliable nieans of collecting informa- 

tlon about the chf^racterlstlt^ ^nd Informatlon-seejclng ^patterns of the * 

users and about the evaluations of users and. reference librarians concern- 
Ingothe reference c-ncoutjjter. 

Scope and Limitations 
A broad view of reference service encompasses all of lt3 slveral facets, 
I.e., the answering of questions, the preparation of bibliographic guides, 
bibliographic Instruction and on-line searching of database. However, this 
study focuses on only pne aspect of reference servlce-.-the ^counter between 
librarian and user resulting from tb-g asking, of a reference question by the 
user. Many reference librarians would consl'det this aspect as the central 
element of reference service; nevertheless, there are other Important ele- 
ments which are not considered in this studyT A ^ 

^Iso^ this study covers .pnly users who come to or phone the Public 

'V 

Documents and Maps Department. It does not consider potential users of ':hHP: 
department. This omission may be an Important one, since the literature on 
government documents stresses their underutlllzatlon. 

There are two methodological limitations related to sampling associated 
with this study: , 

1. The sampling period was limited to several weeks during the spring 
semester, 198A; thus the samples may not be truly represent^lve of the user 
population throughout the year. 

2. The second survey, which lookfe at librarian and user evaluations of 
the reference encounter, was administered to a relatively small sample which 
was not chosen 00 a random basis. It w^s impractical to choose a larger, 
random sample because of limitations on the researcher's time and a desire 
not to further disturb the normal routine of the Department. 
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Survey Procedures 

Survey I 

Subjects - The population for this survey Included users of the Public 
Documents and Maps Department of Perkins Library at Duke University during 
the two-week period of Monday, March 12, 198A - Sunday, March 25, 198A. 

Sample - The sample was scjlected using cluster sampling techniques 
In ^the following manner: 

1) Weekday days (Monday-Thursday) were divided into three time periods 
corresponding to the morrtlng (8 a.m,-l p.m> ), afternoon (1 prm.jr^^5_p_*jiu-)„_^d_ 
evening (7 p .m. -10 p.m.^^ours or the department • Weekend~days~weT^""d^^ 
Into two time periods, inornlng. (8 a,m*-l p.m.) and afternoon (1 p.m. -5 p*m.) 
for Friday and Saturday; and afternoon (2 p.m. -6 p.m.) and evening (6 p.m.-" 
10 p^m.) for Sunday. ^ 

2) Thus thaire were 16 time periods per week and 36 for the 2-week 
"sampHng period. Based on previously compiled dally counts of reference 

questions from the months of October to December, the researcher estimated 

♦ * ■ 

that It was necessary to select 15 of the 36 time periods In order to 
achieve a sample size of at least 100. Then, using a table of random 
numbers, the researcher selected IS time periods. 

Procedure - During each time period selected, the staff member assign- 
ed to the reference desk distributed the questionnaire from the first survey 
to each person who asked a reference question in person; for users who asked 
9 question by telephone the staff member administered the questionnaire on 
the phone. 

Questicnniaire - The questionnaire (see Appendix A) was developed in 
consultation with the staff of the Document*^ Departments After pretesting 
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It with 25 users during the time period of March 5-9, 1984, the researcher 
made minor ijustmento In wording. 

Survey II 

Subjects - The population for this sur/ey Included users of the Public 
Documents and Maps Department during the month of April, 198A, 

Sample and Procedure - Due to limitation on the researcher's time and 
a desire not to disturb further the department's normal work patterns, the 
re&earcher decided against using a rigorous method of. selecting a random 
sample of reference encounters. ^ . 

Instead, the researcher visited the department during afternoons and 

evenings about twice per week during April, While sitting nea^r—^ he-reference 

desk he^observed pat tons andrllbr^rlans and , -based on^ia^experlence i«)xklng_ 

in the department, he selected reference encounter's which seemed typical. 

When the initial encounter between the user and librarian had ended, he gave 

questionnaires to the librarian and the user. 

Questionnaires - The questionnaires (see Appendix B) were the same as 

AO 

those u6ed in the Howell study cited in the previous chapter. Items on the 
two questionnaires were paired so that, both the librarian's and the user's 
evaluations were obtained £ibout each aspect of the reference encounter. 



CHAPTER III 

§ 

PART I - A USER PROFILE OF 
THE PUBLIC DOCUMENTS AND 
r / MAPS DEPARTMENT 

The purpose of the first survey employed In this study was to gather 
baseline data about the characteristics and Information-seeking patterns of 
the users of the Public Documents and Maps Department of Perkins Library at 
Duke University (referred to In this chapter as the Documents Department). 
Tliese data consist of such selected variables as the status of the users 
andthefrequency of flrat-tlme users. 

Two categories of users provide the framework for In-depth analysis: 
undergraduate students and graduate students-faculty. (Graduate students 
and faculty were combined In Oi 2 group based on the assumption that they 
have similar characteristics and Information-seeking patterns.) The 
researcher analyzed group differences using the chl square (X^) test. 
The last section of this part presents the results of this analysis. 

The fallowing sections present the results of the first survey, the^ 
user profile. Each section discusses the question's purpose and the 
results, and presents the data In tabular form. 

Response Rate 

During the two week survey period of March 1? 25, 1984, 127 question- 
naires were distributed to In-house users and five questionnaires were 
administered to telephone callers. Of the 127 in-house questionnaires, 117 
were returned; upon examination one was determined to be unusable. 
All five telephone questionnaires were usable. Thus, a total of 121 

O 1 
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questionnaires were returned resulting In a usable response rate of 
92 percent (n-121). 



Question 1-User Status 
The purpose of this question was to determine who were the most 
frequent users of the Documents Department reference service. As can be 
seen In Table III-l, most users (93%) are members of the Duke community. 
Moreover, most of them (795:) are Duke students, either undergraduate or 
jgraduate. 

Of the nine users not affiliated with Duke, four were graduate students 
from other universities. Thus only five users (4%) could be considered 
members of the "general public" (I.e., people who are not affiliated with 

4 

Duke and who are not students). Faculty members also constitute a 

r^l^^a'^iv^ly small_group_i^^^ 
studies. 







TABLE III - 1 








USER STATUS 




STAT^US 

1 


FREQUENCY 


PERCENTAGE* 


Dukei 
stude 


undergraduate 
nt 


78 




Duke ! 


graduate student 


18 


15% 


Duke 


faculty member 


5 


4% 


Duke 


staff member 

1 _ ■ ' 


8 


7% 


Perkins Library 

staf fi member 

i 


3 


3% 


Other 


1 users 

1 ■ ' 


9 


7% 




TOTAL 


121 


100% 



Percentages have been rounded to whole numbers throughout this study. 
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Quest Ion 2^ ^ Subject Area 
This question's purpose was to discover in what subject areas the users 
were seeking information. Table III-2 shows that moat (73%) of the 121 
respondents were seeking information in the social sciences. Of this group, 
42 (or 35% of the overall total) were working in the field of public policy/ 
political science. This finding reinforces the perception of staff members 
of the Documents Department that documents are heavily used by students 
working on class assignments In the field of public policy/political science. 



Subject Area 

Social Sciences 

Public Policy/ 
political science 

Economics 

History 

Law 

Business 

Other social 
sciences 

Subtotal 



TABLE III - 2 
SUBJECT AREAS OF QUESTIONS 
Frequency 

Ul 

18 
15 

5 

3 

5 



88 



Percentage 

35% 

15% 
12% 

kX 

3% 

4% 

73% 



Non-Social Sciences 

Medicine/ 
health sciences 

Personal need 

Other 



10 
17 



5% 

8% 

U% 



Subtotal 
TOTAL 



33 ^ 27% 

121 100% 
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Question 3 ~ First Visit 
The aim of question 3 was to discover what proportion of the patrons 
were first-time users. Of the 121 users 27 (22%) were visiting the Documents 
Department, for the first time. This figure seems like a high percentage; 
however, since we have no comparative data from other documents or reference 
departments, it is difficult to draw a definite conclusion. Nevertheless, 
it seemR reasonable to assume that these first-time users generally require 
more reference assistance than experienced users. Table III-3 shows the 
proportion of first-time users, 

TABLE III-3 
FIRST VISIT 

Users Frequency Percentage 

First Visit 27 • 22% 

Not first visit 94 • 78% 

TOTAL 121 100% 

Questions A and 5 - Sources of Referral 
The purpose of these questions was to determine how many users, both 
first-time and experienced cnes, were specifically referred to the Documents 
Department and by whom • (Because of the physical location of the Documents 
Department — in the basement of the library — and the specialized nature of 
its resources, it was assumed for these questions that first-time users were 
referred by someone to the department and did not simply decide on their 
own to go there^i) 

Table III - 4 shows that 63 (52%) of the 121 users were referred to the 

Documents Department. For first-time users, the modal source of referral 

♦ 

(56%) was the main Reference Department of Perkins Library. For experienced 

users, 62 percent came to the Documents Department based on their previous 
knowledge of the Department. 
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TABLE III - 4 
SOURCES OF REFERRAL 



First-time 
Users 



Experienced 
Users 



All 
Users 



^^^^^^^^ kL' Frequency Percentage Frequency Percentage Frequency Percentage 



Professor 



112 



IS 



16% 



18 



15% 



Student 



18% 



A% 



8% 



Main 

Reference 
Department 



15 



56% 



lA 



15% 



29 



24% 



Another 
library 



4% 



3% 



3% 



Other 



Total o 
Referrals 



27 



11% 



100% 



36 



38% 



63 



2% 



^2% 



Previous 
knowledge 



58 



62% 



58 



48% 



TOTAL 



27 



100% 



94 



100% 



121 



100% 
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Question 6^ Types of Questions 

Question 2 revealed the users' assessment of the subject areas In which 
Information was soight. The purpose of question 6 was to determine from 
the librarian's viewpf>lnt the types or categories of questions asked. 

The term "type" as used in this context, refers mainly to the search 
strategy used by the librarian in attempting to help the patron find informa- 
tion. In general, questions requiring similar search strategies and the use 
of similar sources were grouped together based on the responses of the users 
to question 6, 

Because the design of the survey instrument did not allow the users to 
de scribe their information desires in depth » each question has been assigned 
to only ont -^tegory. It is likely that some users' actual inf ormation-de- - 
sires overlapped these categories. For such users this categorization repre- 
sents the initial search strategy used for their questions. 

Table III - 5 briefly defines each cat,egory and shows the frequencies 
for each type of question. 



Category 

Legislative 
information 



Request for 
a specific 
title or 
article 



Statistical 
information 



TABLE III ^ 5 

' TYPES OF QUESTIONS 

Definition and Examples of Sources 

Status and background of specific 
legislation; use of CIS Index , 
C ^ Weetxy Report s 

Verifying bibliographic citations 
and finding a Superintendent of 
Documents cliiss number; use of 
OCLC, Monthly Catalog and other 
Indexes, 

Statistics on any subject e.g*, 
crime I health » and population; 
use of American Statistics I\ 
Census materials, and Statist nl 
Abstract. 



Frequency Percentage 
25 21% 



19 



18 



16% 



15% 
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Category 

Current 
topical 
InformatloTi 



Definition and Examples of Sources Frequency 



Maps 



General Information on current topics 
and matters of public policy e.g. 
drugs I U.S. foreign policy* health 
care (inquiry not focused on specific 
legislation); use of vide range of 
indexes and publications. 

ReqjUv'»st for map of a specific area; use 
of aids to locating maps e.g.» map 
catalog » indexes to USGS topographic / 
maps. 

Locating documents older than IS years ^ 
to be used for historical purposes; use 
of various Indexes e.g.* CIS Index to 
Congressional Hearings (prior to 1970) , 
CIS Serial Set Index , Subjec t Index 
1900-1971 to Monthly Catalog . 

International Locatlnj^ information contained primarily 
documents in the publications of international 

— — -"^-^genciesv^^ry~the-yf^l t^d Nat lons^^nd — 
^ The World Bank; use of Index to Inter- 
national Statistics , UN publications and 
Indexes such as UNDOC. * 



Historical 
documents 



Legal 
reference 



Tax 

Information 



Other 



Locating federal and state lavs and 
regulations and Supreme Court cases; use 
of federal and state statutes. Code of 
Federal Regulations U^S» Reports * 

Locating U«S. tax forms and IRS tax 
tax information publications. 

Questions not fitting one of above 
categories. 



13 



Percentage 



8 



TOTAL 



4 

14 
121 



7% 



7% 



5% 



31 

in 

100% 
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Question 7^ Z Sources of 
Bibliographic Citations 
The p'.rpose of question 7 was to find out what percentage of users 
were looking for a specific publication, the title of which they already 
knew before coming to the Documents Department. In addition^ the question 
asked how the readers found a reference to this publication (e.g., through 
an index or a reference in a book) . 
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After analyzing the responses to this question, the researcher 
decided that It had been widely misunderstood. Of the 121 respondents, 
76 (633;) indicated that they were looking for a specific publication 
whose title they knew beforehand. In analyzing the responses to question 
6, however, the researcher found that only 19 (16%) of the respondents 
Indicated that the "information" they wanted was retrieval of a specific 
title. It seems that many users who answered question 7 had some Idea 
that there was a specific publication which would supply the information 
they wanted, but the answers to question 6 Indicated that many fewer users 
actually knew the title of this publication. * 

Thus, for question 7, the tesearcher decided to analyze only the 
responses of those users whose answers to question 6 clearly indicated they 

« 

came to the department to retrieve a specific publication. whose title they 



knew beforehand. 

Table III - 6 presents the sources of bibliographic citations for 
the titles desired by these 19 users. 



TABLE III - 6 
SOURCES OF BIBLIOGRAPHIC CITATIONS 



Source 

Reference In a book, 
Journal or newspaper 

Professor 

Index other than those In 
the Documents Department 

Main Catalog. 

Student 

Other 

TOTAL 



Frequency 

5 

A 
3 

2 
2 
3 

19 \ 



Percentage 

272 

21Z 
16% 

10% 
10% 

« 

16% 



. 100'/ 
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In order to have avoided confusion about question 7, It would have 
been desirable to ask the user to write the title of ihe publication 
being sought as part of the answer to the question. This might have 
confirmed that the user had a specific title in mind. 
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Question B - Amount of Time 
Spent In the Documents Department 
This questioned purpose was to determine how much time users spent 
in the Documents Department. As Illustrated In Table III - 7, most users 
(722) spent less than half an hour, although a substantial proportion (282) 
spent more time. Of the six time periods, the modal group (28^) Is the 
5-15 minute group. 

As with question 3 (which identified first-time users). It is difficult 
to draw any firm conclusions from this information since we have no standards 



for comparison. The data suggest, however, that many users (those who stay 

more than half an hour) have complex reference questions which may require 

in-depth assistance from a librarian. 

TABLE III - 7 
AMOUNT OF TIME SPENT 
IN THE DOCUMENTS DEPARTMENT 



Time Period 

Less than 5 minutes 

5-15 minutes 

16-30 minutes 

subtotal- 
's hour or less 

3-1-59 minutes 

lr2 hours 

M ore than 2^ hours 

subtotal - more 
than h >ur 

■ TOTAL ' 



Frequency 
25 

34 

28 

87 

14 
14 
6 



Percentage 
211 

28% 

23X 

11% 

111 

ni' 

42 



34 



282 



121 



1002 



37 



/ 



\ i 
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"^ ^Analysis of Group Differences ^ ^ 

Two major groups of users, i.e., Duke undergraduate students and 1 
Duke graduate students-faculty, were selected for analysis of group \ 

differences In characteristics and Information-seeking patterns. (It was 

ft . > i, 

not feasible to analyze other groups because of the low numbers of them \ 
Included In the sample.) 

Table III - 8 shows the frequency distribution of undergraduate 
students and graduate 8tuden*:8~f acuity In the sample. 

TABLE III - 8 
USER STATUS 
Frequeng^ 
78 



Status 

Duke undergraduate 
students 

-Duke-graduate^ 
student-faculty 

Other 



Percentage 
642 



— 



23 



20 



17% 



TOTAL 121 ^ . loor 

Potential group dlfferences_between undergraduate students and 
graduate students-faculty were teste! for using X^. No^ statistically 
significant differences between the two groups were found In any of the 
variables, including subject areas of questions, proportion of first-time 
users, sources of referral, types of questions, and sources of bibliographic 
citations. 



ERIC 



Summary 

Most users of the Documents Department reference service are Duke 
students. Also, most are seeking Information In the area of the social 
sciences, ^bout half the users are referred to the department, often by 
the Main Reference Department of the library. The most frequently asked 

38 
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types of questions are 1) legislative information, 2) request for a spe- 
cific title or article, and 3) statistical information. For the variables 
tested, no statistically significant differences emerged w^^th respect to 
characteristics or information-seeking patterns between the groups of 
undergraduate students and graduat'e students-faculty. 

• . 

PART 2 - AN ANALYSIS OF 
USER - LIBRARIAN REFERENCE 
• , ENCOUNTERS 

The purpose, of the second survey was to gain some insight into the 
effectiveness of the provision of reference service in the Documents 
Department by analyzing a series of reference encounters between users and 
librarians. 

7' surv^ consisted of two questionnaires-one for librarians and one 

for users. The Items on each questionnaire were matched so as to obtain 
both the librarian's and the user's evaluations of each aspect of the 
reference encounter. Tlie researcher analyzed group differences in response 
patterns using the chl sqtiare (X') method at the .05 level of significance. 

The following sections discuss the purpose and results of each 
question and, where appropriate, present the results in tabular form. 

Response Rate 

During the survey period of April 198A, the researcher asked AO users 
to participate in the .survey. None refused and all returned their question- 
nalres, as did the librarians, resulting in a 100% response rate.' 

Question \_ Question Negotiation ^ 

The researcher reviewed the librarians' questionnaires to jd^termine 

whether there was a difference between the responses to the twjoj |)arts of 

question 1 - "the user's original question" and "what the user really • 

• • ' i 

II * 
wanted. In the case of substantially different responses, the researcher 

assumed £har the librarian usecf question negotiation to clarify what 



Information the user actually wanted. Of the 40 questions, 16 (A0%) 
required question negotiation. 

Question 2 Z ^^^^ Status 
The purpose of this question was to determine the acc;:r«w/ of the 
librarians* initial perceptions of user status. Such a perception is 
important, because an accurate impression of user status can help the 
'librarian determine the needs of the user. For example, undergraduates 
working on a term paper have different information needs than graduate 
students working on a doctoral thesis. Librarian perceptions of user 
status were correct In 35 (SB%) of AO cases. This seems 2Jkk a high 
rate of accuracy and suggests that the librarians are generally very 
familiar with tlie various groups of i^^sero^ A - 

Table III - 9 shows the actual ^status of the users 
studied In this survey. The distribution of user groups of this sample, 
which was not chosen on a random baslsi is roughly the same as the dlstri 

bution of the groups in the sample for the flvst survey, which was^chosen 

■ ' 'A 

on a random basis. Thus It appears thatv^!e>s,^j)le used for this survey 
may be representative of the user population. . 

- ^ ' TABLE 111-9 



USER STATUS 



Status Frequency Percentage 

Duke undergraduate 24 60% 
student 

Duke graduate student." 7 18% 

Duke faculty member 2 ' 5% . 

Duke staff member • 1 2% 

Perkins Library i 2% 

staff member * 

Other 13% 

• ' '■ ' ' 

TOTAL 40 ' 100% 
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Question 2 z Familiarity 

with the Docutnents Department 

As a reference librarian begins to assist h. patron, it Is useful to 

"•••determine how much the patron already knows about the reference tools 

'I 

and the library In order to choose an appropriate technical level for his 
Interaction with the patron^" 

Users and librarians were in agreement in their ratings of user familiar- 
Ity with the Documents Department in 21 out of 40 cases. Users and llbrarlana 
concurred that the user's familiarity was "poor" or "fair" in/19 cases, and 
"good"^4n only 2 cases. In the 19 cases of disagreeraen,t , the user rated his 

familiarity with the Department higher than the librarian did in 13 cases and 

■ ' ft 

lower in only 6 cases. 

table 1X1 ^ 10 presents the frequency distribution of the evaluations of 

user familiarity with the Doc«nents Department, As shown by the value, 

there is a statistically significant difference h:tween upers and librarians 

i 
f 

in the overall pattern of responses, i.e., the users tend to rate their 
familiarity higher than the librarians do. It is likely, however, that the 
librarians* evaluations are more realistic. 

TABLE III - 10 

USER FAMILIARITY WITl/ 
THE DOCUMENTS DEPARTMSkT- 
ALL RESPONSES / 

-I ^■ 
Rating of By Librarians ' B^ Users 

User Finnlliarlt: y F requency Percentage , Frequency Percentap^e 

good , 3 8% } 10 25% 

fair or poor 37 923{' 30 75% 



TOTAI^ 40 100% I 100% 

« A. 52 with 1 d.f, [ 

t 

Significance « .05 f 
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When only cases, of non-consensus between users and librarians are 
considered there is also a statistically significant difference. Table 
III - II shows the frequency distribution of responses for cares of non- 
consensus and the value of 

^ TABLE III - II 

USER FAMILIARITY - 
CAPES OF NON-CONSENSUS 

Rating of Rating by librarian , .Rating by user 
User F amiliarity 

Higher 6 13 = 

Lower 13 6 

TOTAL 19 19 

» 5.16 with I d.f* J 
Significance « .05 

Thus Ir is possible to reject the first null hypothesis, l»e., that 
there is no statistically significant difference between users and librar- 
ians in their evaluations of user familiarity with the Documents Department. 

Quesiton 4 - Pinpointing the 
User's Needs 

The purpose of this question was to measur^ user statisf action with 
one aspect of the reference encounter ^ the ability of the librarian to 
pinpoint the information wanted by the user. As noted It) Chapter I, how- 
ever, nmasures of user satisfaction vlth library services tend to be high, 
possibly due to the users' low expectations of reference service or to 
their •difficulty distinguishing between their pleasure at being helped 
and their satisfaction with the quality of help being received." On the 
other hand, librarians may have a tendency to rate their performance unreal- 

istlcally low. Thus it is appropriate to consider both user and librarian 
measures of librarian performance. 
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Users and librarians agreed In 23 cases that the librarians had pin- 
pointed the users' information needs "very well," and In 3 eases that the 
librarians had done "moderately well." In the U cases of non-consensus, 
the llbrarlnn rated his or her performance lower than the user did in 12 
cases and higher in only 2 cases. 

Table III - 12 shows the evaluations of librarians and usevs of the 
librarian's performance in- pinpointing the user's information needs. The 

value indicates that there is a statistically significant difference 
between users and llbrarlanp in their rating of librarian performance i.e.. 
the users tend to rate the librarians' performance higher than do the 
librarians. 



R ating of 

Librarian Performance 



Very Well 

Moderately well 
or poorly 



TABLE III - 12 

PINPOINTING USER 
INFORMATION NEEDS- 
ALL RESPONSES 

By Librarians 



Frequency Percentage 
, 25 63% 
15 37% 



By Users 

Frequency Percentage 
88Z 

5 \ 12% 



TOTAL 

« 6.67 with 1 d.f. 
Significance « .01 



40 



100% 



40 



100% 



When only cases of non-consensus are considered, there is also a stat- 
istically significant difference between the ratings of users and librarians. 
Table III - 13 presents the frequency distribution of responses for cases of 
non-consensus and the value of X^. 
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TABLE III- 13 

PINPOINTING USER 
INFORMATION NEEDS- 
CASES OF NON-CONSENSUS 

Ratings By Librarians By Users 

Librarian Performance 

Higher 2 12 

Lower ' 12 2 

TOTAL U 14 

X* « U.28 with 1 d.f . 

Significance >> .01 

Thus it is possible to reject the second null hypothesis, i.e. 9 that 
there Is no statistically significant difference .between users and 
librarians in their evaluations of the librarian's ability co pinpoint the 
user's infonaatlon needs. 



Question 5^ ^ Supplying the 
R^.ght Amount ' of Information 

As with the previous question, the purpose of this question was to 

' ^\ ■ ■ 

measure an aspect of user satisf action^ in this ^ase the performance of 

the librarian in supplying the right amount of Information, 

* \ 

Users and librarians concurred in 26 ot^t of AO cases that the user 

\ 

received about the right amount of informatiot^t They agreed in two cases 
that the user received less Information than he^deslred^ and In one case 
that the user received more information than he paired* In the 11 casee 
of non-consensus » the librarian reported supplying less information than 
the user reported receiving in 7 cases and more information in A cases. 

Table III - U presents the frequency ulstributlon of the evaluations 
by users and librarians of the librarians' performance in supplying the 
right amount of infoYmation, In contrast to the results of question A, 
there is no. statistically significant dlffetence between the responses of 

44 ■ 
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all users and all librarians. In addition, there Is no statistically 
significant difference between the responses of users and librarians in 
only those cases of non-consensus. Thus it Is not possible to reject the 
third null hypothesis, i.e., that there Is no statistically significant 
difference between users and librarians In their evaluations of the librar- 
ian's ability to supply, the right amount of Information. 



Rating of 
Librarian 
Performance 

Right amount of 
Information provided 

Too little or too 
much Information 
provided 

TOTAI, 



TABLE III - U 

SUPPLYING THE RIGHT AMOUNT 
OF INFORMATION- 
ALL RESPONSES. 

By Librarians 

Frequency Percentage 

30 151 



10 



25% 



By Users 
Frequency Percentage 
33 83% 

7 17% 



AO 



100% 



AO 100% 

It Is difficult to explain for this question why librarians did not 
rate their performance lower than the users did, as happened with the re- 
sults of question A. It is possible that the number of cases in the sample 
was too small to reveal a difference. Or It may be that the librarian's 
confidence in his or her performance increases once he or she feels the 
user's needs have been pinpointed. 



Question 6 - Instruction in the 
Use of Reference Tools 

The purpose of this question was to discover how well the users 
^^elved inteided instruction by the librarians In the use of reference 
tools. 

Librarians and patrons agreed in 26 cases that instruction was given 
and in 6 cases that it had not been given. In the 8 cnses of non-consensus 
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7 users perceived instruction when none had been Intended, and one did 



not perceive intended instruction. ' $ , 

Table III - 15 presents the frequency distribution of librarian and user 
responses to question 6. There Is no statistically significant difference 
between the two groups. Thus It is not possible to reject the fourth null 
hypothesis, I.e., that is no statistically significant difference between 
users and librarians In their awareness of intended instruction. 



Instruction 
Intended 



yes 



no 



TOTAL 



TABLE III - 15 • 

AWARENESS OF INTENDED 
INSTRUCTION- 
ALL RESPONSES 

Intended 



by Librarians 
Frequency Percentage 



27 
13 



40 



68!i; 
32:? 

Too%' 



Awareness 
by Users 
Frequency Percentage 



33 



833; 



17% 



40 



100% 



The analysis of question 6 may be Mora enlightening when viewed In 
conjunction with the results pf question 5. For example. In 24 of the 26 • 
cases of consensus on the presence of Instruction, the user expressed 
satisfaction with the amount of information received. Furthermore, of the 
7 users from question 5 who said they received too little or too much 
information, there was only one case of consensus that instruction had 
taken place. 

These findings are similar to those of the Howell study where accord- 
ing to the authors: 

This suggests that reference librarians can assist patrons more 
effectively when they consciously cultivate a teaching role as 
opposed to acting more passively (and perhaps m5Ve impersonally) 
as an information source. , 



A2 

Summary of Hypotheses and Results 
The first two null hypotheses, I.e., that there Is no statistically 
significant differences between the users' and librarians' evaluations of 
1) the user's knowledge of government documents and 2) the librarian's 
ability to pinpoint the user's information needs were rejected. The third 
and fourth null hypotheses > however, were not disproved. These were that 
3). there is no statistically significant difference between the evaluations 
of users And librarians of the librarians 's performance in supplying the 
right amount of information and 4) there is no statistically significant 
difference between the users and librarians In their awareness of intended 
Instruction. 

In general, these findings are similar to, but not exactly the same 
as, those of the Howell study. One difference Is th&t this study found that 
users tend to rate their familiarity with documents higher than librarians . 
do, while the Howell study found no difference between the groups for this 
variable. This finding suggests that the maay users of the Documents 
Department may have both an inadequate knowledge of government documents 
and an unawareness of their lack of knowledge. 

For the two measures of patron satisfaction (hypotheses 3 and A), the 
Howell study revealed the users' ratings to be significantly higher than the 
librarians' ratings for both measures. This study, however, found a 
significant difference for the first measure only (hypothesis 3). These 
findings suggest that the most difficult part of the reference encounter for 
the librarians is pinpontlng the Information needs of the users. Such a 
conclusion is reinforced by the librarians' low rating of user familiarity 
with government dod^ments. In other words, it is more difficult to clarify 
what a user wants when the user has a poor knowledge of both what Information 
might be available and how to find It. 
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The results for the fourth hypothesis are perhaps the most significant. 
As the Howell study found » users are generally well aware of Intended instruc- 
tlon by the librarians In the use of reference tools. Moreover, the user Is 
more often satisfied with the outcome of the reference encounter when there Is 
consensus that Instruction has taken place than when there Is disagreement i 
about the presence of Instruction, 

The concluding chapter of this study discusses further the implications 
of the results of both the first and second surveys. 
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CHAPTER IV 
CONCLUSION AND RECOMMENDATIONS 

The results of this studr seem to be generally comparable with the 
findings of other studies, In particular the Howell Study. User satis- 
faction with the provision of reference service in the Department appears 
to be high, although user familiarity with documents reference tools and 
sources appeatrs to be low. In addition, users tend to respond favoriibly 
to individual instruction by the librarian in the use of reference tools, 
and sourtes. Thus this study has essentially replicated Howell's study 
for the case of reference service for government documents. 

However, because this study was concerned with only one institution 
and because of the methodological limitations of the surveys, this study's 
results are not generalizable for all government documents depart^ments. 
Thus there is a need for other researchers to generate comparative data 
for other documents departments. For example, it would be useful to 
learn whether the user profile of other documents departments In large 
academic depository libraries resembles that of Duke. 

The results of this study, however, do have further implications for 
the Documents Department at Duke. The following sections provide this 
researcher's recommendations concerning the provision of reference service 
in the Documents Department* 

Staff Members' Knowledge and Reference Skills 
As noted by Hernon and McClure, it is likely that "...the individual 

If 

library staff member is the single most significant factor affecting the 
quality of reference service for government documents," The results of 
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both the first and second surveys point to several areas where the staff ^ 
members of the department might concentrate their efforts to improve their 
knowledge and skills. 

From the results of. question 2 of the first survey (subject areas of 
questions) it Is clear that a good knowledge U the social sciences, parti- 
cularly in public policy/political science, is an asset for doing documents 
reference work. Furthermore, the results of question 6 (types of questions 
demonstrate that good skills In using certain heavily used indexes and 
sources (e.g., CIS Index , Congressional Quarterly publications, ASI Index ) 
will allow the reference staff member to handle a large volume of the 
reference questions In the Department. Administrators pf the Department 
might, consider these factors In selecting and training the non-professional 
staff members of the Documents Department. 

The results of the second survey underline the Importance of the . 
reference Interview. It seems that many users, perhaps because of their 
unfamiliarity with the Department, express poorly what information they 
want and are also unaware of what Information is available. T^ius It 
would be desirable for the reference staff to evaluate and Improve their 
Interpersonal communication skills. As shown by the results of questions 
5 and 6 of the second survey, good comniunlcation between user and librarian 
seems to lead more often to user satisfaction. Hernpn and McClure suggest 
a number of strategies for improving the process of the reference 
interview. 

Blbllog.raphic Instruction 
The Documents Department currently has an active program of biblio- 
graphic instruction which Is well-received by the user community.^ Most 
sessions are set up at the request of a faculty member. Based on the 
results of this study, however, the Documents Department might consider 
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a more fcrmallzed program. For example, Just as the Reference Department 
gives a formal introduction to all freshmen on library research techniques, 
so too might the Documents Department set up a general Introduction to 
documents research techniques for^U^ public policy/political science 
students. 

• Given the current level of staffing In the Departient, such a program 
might not be feasible. It should be possible, however, to Identify certain 
faculty members whose classes frequently use the Department and then ensure 
ihlat all such classes receive formal Instruction from a member of the 
Documents staff. 

Further Evaluation of Reference Service 

Wltlle this study has provided sc^e information about the effective- 
ness of tlvfc Department's reference service, it would be desirable to design 
\ a methodology for continuing evaluation of the Department's reference 
service. As noted by Lancaster, such evaluation should be used as a 
diagnostic tcol" and not as a part of evaluations for salary increases 
or promotion. 

; \ 

OneXapproach to evaluation would be the administration of test 

• \ ' ■ •, - ; 

questions, either directly or unobtrusively, to the reference staff. 

\ ■ . 

•As discussed l^n Chapter I, this Is the most reliable and direct method of 

assessing the performance of the reference staff. Such quesltons would 
^^rhelp to reveal weak esses in staff members' knowledge of subjects and 
sources, \ 

Further surveys of the users might also aid in the evaluation 
process. Despite the tendency of users to give high performance ratings 
to librarian^, it might be possible to design survey instruments that 
are more sensitive to the users' s true levels of satisfaction. 
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Questions aimed discovering patron expectations for service 
and forcing pat/one to make the distinction between the 
psychological and substantive outcome of reference encounters 
would be useful additions to future patron-sat Is fact Ion. survey 
Instruments. 



In addition, since , this study has surveyed only actual users of the 
Department. It would be desirable to survey the potential user community, 
e.g. all students and faculty In the social sciences. , 

Cooperation and. Referral 
demonstrated in the results of the first survey, librarians" In 
the Reference Department along with faculty are important sources 
of referrals of patrons to the Documents Department. While there is no 
indication of any problems associated with Aferrals from these sources, 
it /Would be desirable for the Documents Department to help other 
librarians and professors better understand what kinds of information 
are available in government documents. 

An example of formal cooperation between the Documents Department 
and the Reference Department occurred during one semester last year when 
a librarian from the Reference Department spent several hours per w^ek 
learning about documents and stiffing the reference desk In the 
Documents Department. Ideally, such exchanges should work both ways, 
since librarians In the Documents Department often refer patrons, to the 
Reference Department* 

Because of their low usage of documents and their many other time 
commitments. It is more difficult to Increase facility members* awareness 
and. knowledge of government documents. In addition, many jf acuity - 
members' knowledge and awareness of library resources in general may be 
relatively low. Thus it would be desirable. If staffing and time permit, 
for the Documents Department tp t^st an "outreach" program aimed at 



A8 

Increasing faculty awareness of government documents; or if the library 
administration should undertake a plan to improve .faculty awareness of 
library resources then government documents should be specifically 
included. 

Areas for Further R esearch 
As already noted, ^ome of the results of this study would be more 
meaningful If comparative data were available In other studies. There 
Is a need for comparative data about the characteristics and information- 
^seeking patterns^ of users of other government documents departments. 
It would also be valuable to further test other methodologies for the 
evaluation of reference service such as the administration of test 
questions. * 

In order to discover what are the unique aspects of docupients refer- 
ence service, there is also a need to compa^re the provision of reference 
serv)lce in documents departments to its provision in general referenc\ 
departments.. For example, does the search strategy for documents reference 
generally require more steps than for general reference? And how do the 
proportions of factual vs. l1i-depth questions compare between documents 
reference and- general reference? 

* 4 

Hernon and McClure suggest a number of ottier areas where research 
on the provisloi;! of reference for government documents is needed. As 
Hernon and McClure conclude, the v^lue of such research lies in its 

potential for improving the quality, of reference service for government 

r ' 

documents: 

Awareness of the role of research, combined with implementation of 
research results and development studies, will enable documents 
librarians to adapt better to new environments and situations, as 
well as to better serve the-lnformatlon needs of their clientele, 
both present and potential. 
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APPENDIX A 

DUKE UNIVERSITY LIBRARY 
PUBLIC DOCUMEN"^S AND MAPS DEPARTMENT USER SUliVEY 



Date:' 
Time :" 



In order to learn more abotit the users of the Public Documents and- 
Maps Department and thus lmprove\ our r^ferenqe service, we are conducting 
this survey. Please take a few minutes to complete this questionnaire, 
and leave it in the box at the reference desk before you leave the depart- 
inent. Thank you. ^ v . 



1 



5. 



Are you (please check the appropriate category.) 

^Duke S,taff Member? 

Other? 



Duke undergraduate student? 

professi< 
(Specify 



Duke graduate or professional 
school student? 
dept. or school.) 



Duke faculty member? 
dept. or school.) 



(Specify 



Specify occupation 
and institutional affiliation, 
e.g. ,' graduate student at UNC 
or engineer at IBM, Raleigh. 



_Perkins library staff member? 



For what subject area or course are you seeking information?, (Please 
check the most appropriate category.) 

_Public Policy \- ^Medicine/Health Science* _ 

^Other (Specify subject.) . 



_Political Science 
^[History 
^Economics 
_Business 
Law 



JPersonal need, not related to 
school or work. (Specify.) 



Is this your first visit to the Public Documents and Maps Department? 

_YES. IF^YES, ANSWER QUESTION 4 AND SKIP QUESTION 5. 

NO 1/ NO, ANSWER QUESTION 5 AND SKIP QUESTION 4. 



If this Is your/ first visit to the Public Documents and Maps Department.' how 
did you find out about it. 

— Referred here by a professor ^Referred hete by a library 

^Referred here by a student staff* member In another library 

' ^Referred here by a librarian (Specify library.)" 

in the main Reference Dept. . 

O ther (Specify.) ~~~~~T 



upstairs. 



If this is not your first visit to the Public Documents and Maps Depart- 
ment, who referred you here this time? 



_A professor 
_A student 

^A librarian in the main 
Reference Dept. upstairs 

_A librarian In another library 
(Specify library.) 



Other (Specify.) 



_No one. I came here based on 
previous knowledge or exper- 
ience with the department. 



ERIC 



54 



50 



briefly describe the information you\ ite looking for, e.g., the consumer 
price index for 1983, a map of Lebanon, or information about a senator's 
voting record.^ 



If you came to the department lookinglfor a specific publication, the 
title of which you knew^ beforehand, wl)ere\did you get the reference? 

^An index (Specify index. )_ ' ' ^ \ ■ 

^Main catalog, upstairs. '. ^ t 



Reference in a book, journal, pr newspaper, 
_Reference ^:n another government document .\ 

_From a professor or graduate assistant. \ 
_From a student 

Other (Speciify.) 



Estimate the length of time you spent in the Publi^c Documents and Mahs 
Department during this visit. 

, less t'rtan 5 minutes 31-59 minutes 

5-15 minutes , ]|l-2 hours 

16-30 minutes ^more than 2 hours 



/ 



\ 



\ • 

\ 

\ 

\ 
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DUKE UNIVERSITY 
PUBLIC DOCUMENTS AND MAPS DEPARTMENT 
USER SURVEY II, STAFF COPY 



No.:_ 
Date:] 
Time 



1, What was the user's orlgl^ial question? 
What did the us^r really want? 

2. On the basis o|f this encounter, I would guess that this user is; 

puke Undergraduate student 

Duke graduate student 



J)uke faculty member 

J>uke staff member 

^Perklt^s Library staff member 

"othe r / 



3. On the basis of this encounter, I would Judge this user^s level of 
familiarity with the Documents DeparY.men*- ^nd its resources to be: 

good 
|. ^ fair 
poor 



4. During the question-negotiation process, how well did' you pinpoint the^ 
patVon'^ needs? 



very well 




\ moderately well 




\ poorly 




CommeVits: 




\- . ■ ■ - • \ 


\i / 
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gpt less information than he or she wanted? 
wAs satisfied with the amount of information? 
ot more information that he or she wr^nted? 



6, While assisting the user, did you instruct him or her in the use of 
reference sources or tools, Indexes, etc.? 



^no 



[If yes, which ones? 



(SprinR, 1984) \ 
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DUKE UNIVERSITY - 
PUBLIC DOCUMENTS AND MAPS DEPARTMENT. 
, USER SURVEY II, USER COPY - 

li No.: 
I • Date: 

Time ; 

!• What did you ask the reference librarian? 



2. Are you (Please check the appropriate category.) 
D uke undergraduate student? 



Duke graduate student? 
Duke faculty member? 
J)iike staff member? , 
Perkins Library staff member? 

_Other? (Please specify occupation and institutional affiliation, 
e.g», grad. student at UNC or engineer at IBM) 



3. How would you rate your familiarity with the Documents Department? 

g ood 
f^lr 



^poor \ 

\ 

4. How well did the reference librarian pinpoint your needs? 

very well 

moderately well 

^poorly 



Comments: 



5. Did you receive (check one) 



too little information? 

about the right amount of information? ^ 
more Information that you wanted or could use? 



0 



6. While Assisting you, did the librarian instruct you In the use of 
reference sources or tools, such as the card catalog, bibliographic 
Indexei^, 0!tc»? 

y es no If yes, what did you learn? 



(Spring, 1984) 
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